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(54) Method and apparatus for providing interactive services with multiple interfaces 



(57) Interactive services with multiple interfaces are 
realized by employing a modular approach to their 
implementation. Such an approach facilitates support- 
ing natural language understanding interaction with 
users through use of interfaces that at least allow the 
user to provide information beyond what is currently 
being requested by the service, and/or different order- 
ing of inputs, and/or incomplete information, and/or cor- 
rection of information, and/or the return of control to 
prior points in the service. This is realized, in an embod- 
iment of the invention, by employing an interactive serv- 
ice logic that uses "reactive constraint graphs", i.e., a 
form of event-driven graph, in which nodes contain con- 
tracts on events, in conjunction with a service monitor. 
The service monitor manages the communication 
between the service logic and the multiple user inter- 
faces. As such it provides a communication mechanism 
in the form of a so-called "look ahead" in which the user 
can input information beyond what is currently required 
by the interactive service. Specifically, in each round of 
user interaction, reactive constraint graphs report to the 
service monitor the set of events that are currently ena- 
bled. These enable events correspond to the informa- 
tion the interactive service is currently requesting (or 
ready to accept) from a user. The user interfaces collect 
information (perhaps beyond this set) from the user and 
send it to the service monitor. The service monitor then 
matches the received information against the informa- 



tion requested by the service logic, and sends the infor- 
mation one event at a time to the service logic, in a 
priority order specified by the service provider. After 
each event is sent to the service logic from the service 
monitor, the service logic reports its new set of enabled 
events back to the service monitor. In turn, the service 
monitor sends another event that matches the enabled 
events to the service logic. The service monitor then 
notifies the user interfaces to send a new collection, i.e., 
set, of information from the individual users only when 
no current information matches the enabled events. 
This approach advantageously allows a user to input 
information beyond what is currently needed by the 
interactive service. 
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Description 
Related Application 

[0001] United States patent application Serial No. 5 
(T. J. Ball et al. Case 10-4-2-1-9-7) was filed concur- 
rently herewith. 

Technical Field 

[0002] This invention relates to providing interactive 
services and, more particularly, to interactive services 
with multiple interfaces. 

Background of the Invention 

[0003] Prior arrangements for implementing inter- 
active services with multiple interfaces employed finite- 
state machines (FSM) to realize the necessary service 
logic. This was achieved either explicitly using FSM 
tools such as ObjecTime or StateCharts, or implicitly 
using programming languages/technologies such as C, 
C++, Perl or Microsoft Active Server Pages (ASP). 
[0004] In order to support natural language under- 
standing interaction with users, services with multiple 
flexible interfaces need to allow the user to provide infor- 
mation beyond what is currently being requested by the 
service, different ordering of inputs, incomplete informa- 
tion, correction of information and the return of control 
to previous points in the service. Finite-state machine 
based approaches, however, cannot support these 
desired aspects in a modular fashion. Indeed, every 
possible ordering of inputs needs to be explicitly speci- 
fied, resulting in an exponentially increasing number of 
states and transitions in the FSM. Additionally, including 
the other desirable aspects noted above further 
increases the size and complexity of the FSM. Moreo- 
ver, it is extremely difficult to modify the FSM; for exam- 
ple, the number of modifications required to the FSM to 
add another input can be exponential in the size of the 
input set. 

[0005] User interaction with the prior interactive 
services is in so-called rounds. During each round the 
user must provide exactly the information requested by 
the service in its then current state. Specifically, in a 
graphical interface, the user must click on buttons corre- 
sponding to input choices; in a web-based interface, the 
user must click on buttons or fill in text fields; in a touch- 
tone telephone interface, the user must press buttons of 
the key pad; and in an automatic speech recognition 
based interface, the user must utter exactly the informa- 
tion that the service requires in its current state. Such 
arrangements are very restrictive in requiring the 
exactly needed information. 

[0006] Prior arrangements employed to provide 
spoken dialog interaction are focused on a single, spo- 
ken language interface, which is limiting. 



Summary Of The Invention 

[0007] Problems and limitations of prior known 
arrangements for providing interactive services are 
overcome by employing a modular approach to imple- 
menting interactive services with a plurality, i.e., multi- 
ple, user interfaces. Such an approach facilitates 
supporting natural language understanding interaction 
with users through use of user interfaces that at least 
allow the user to provide information beyond what is 
currently being requested by the service, and/or differ- 
ent ordering of inputs from the user interfaces, and/or 
incomplete information from the user interfaces, and/or 
correction of information from the user interfaces, 
and/or the return of control to prior points in the service 
in response to a request from at least one of the plurality 
of user interfaces. 

[0008] This is realized, in an embodiment of the 
invention, by employing an interactive service logic that 
uses "reactive constraint graphs", i.e., a form of event- 
driven graph in which nodes contain constraints on 
events, in conjunction with a service monitor. The serv- 
ice monitor manages the communication between the 
service logic and the multiple user interfaces. As such it 
provides a communication mechanism in the form of a 
so-called "look ahead" in which the user can input infor- 
mation beyond what is currently required by the interac- 
tive service. 

[0009] Specifically, in each round of user interac- 
tion, reactive constraint graphs report to the service 
monitor the set of events that are currently enabled. 
These enabled events correspond to the information the 
interactive service is currently requesting (or ready to 
accept) from a user. The user interfaces collect informa- 
tion (perhaps beyond this set) from the user and send it 
to the service monitor. The service monitor then 
matches the received information against the informa- 
tion requested by the service logic, and sends the infor- 
mation one event at a time to the service logic, in a 
priority order specified by the service provider. After 
each event is sent to the service logic from the service 
monitor, the service logic reports its new set of enabled 
events back to the service monitor. In turn, the service 
monitor sends another event that matches the enabled 
events to the service logic. The service monitor then 
notifies the user interfaces to send a new collection, i.e., 
set, of information from the individual users only when 
no current information matches the enabled events. 
This approach advantageously allows a user to input 
information beyond what is currently needed by the 
interactive service. 

Brief Description Of The Drawing 

[0010] 

FIG. 1 shows, in simplified block diagram form, 
details of an interactive system in which the inven- 
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tion may be advantageously employed; 

FIG. 2 shows, in simplified block form, details of the 
SISL (Several Interfaces, Single Logic) architecture 
employed in the embodiment of FIG. 1 ; 5 

FIG. 3 is a flow chart illustrating the steps imple- 
mented by the service monitor in accordance with 
the invention; 

10 

FIG. 4 is a flow diagram illustrating a reactive con- 
straint graph for a portion of an Any-Time Teller 
banking service example; and 

FIG. 5 illustrates the steps performed in an example 15 
execution of the AnyTime Teller including look 
ahead in accordance with the invention. 



Detailed Description 

20 

[0011] FIG. 1 shows, in simplified block diagram 
form, details of an interactive system in which the inven- 
tion may be advantageously employed. It should be 
noted that the arrangement shown in FIG. 1 is but one 
example of an application of the invention. Indeed, a 25 
plurality of different user interfaces and/or one more 
identical user interfaces may be employed as desired. 
[0012] Specifically, shown in FIG. 1 is SISL (Several 
Interfaces, Single Logic) service unit 101; home/office 
computer 1 02 used as a customer and/or provider inter- 30 
face and includes automatic speech recognition includ- 
ing natural language understanding, if desired, that is 
interfaced to SISL service unit 101 via an Internet link 
1 03; telephone 1 04 also used as a customer and/or pro- 
vider interface that is interfaced to SISL service unit 1 01 35 
via a telephone network 105 including, for example, 
touch-tone, i.e., multi -frequency, signaling; computer 
1 06 used as a customer and/or provider interface, which 
may also include a natural language understanding, 
that is interfaced to SISL service unit 101 via a local 40 
area network (LAN); and ATM (automatic teller 
machine) used as a customer interface and, typically, is 
interfaced to SISL service unit 1 01 via a direct connect 
109. A key advantage of SISL is that all user interfaces 
to a service share the same service logic. SISL provides 45 
a clean separation between the service logic and the 
software for a variety of user interfaces including but not 
limited to Java applets, HTML pages, speech-based 
natural language dialog, and telephone-based voice 
access. In this example, SISL is implemented using the so 
JAVA programming language. 

[0013] At the outset it is felt best to describe some 
of the principles employed in implementing the flexible 
interactive service including an embodiment of the 
invention. For simplicity and clarity of exposition, these 55 
principles will be presented in the context of a so-called 
Any-Time Teller banking service employing the inven- 
tion. The Any-Time Teller is an interactive banking serv- 



ice. The service is login protected; customers must 
authenticate themselves by entering an identifier (login) 
and PIN, i.e., personal identification number, (pass- 
word) to access the functions. As customers may have 
many money accounts, most functions require the cus- 
tomer to select the account(s) involved. Once authenti- 
cated, the customer may: 

• Make a deposit. 

• Make a withdrawal. The service makes sure the 
customer has enough money in the account, then 
withdraws the specified amount. 

• Transfer funds between accounts. The service 
prompts the customer to select a source and target 
account, and a transfer amount, and performs the 
transfer if (1) the customer has enough money in 
the source account, and (2) transfers are permitted 
between the two accounts. 

• Get the balance of an account. Display the balance 
with respect to all posted transactions. 

• View the transactions of an account. Display the 
transactions for the selected account. 

• See the last transaction. Display the last action the 
customer performed. 

[001 4] The transfer capability of the Any-Time Teller 
requires the collection of three events: the source 
account (src), target account (tgt), and dollar amount 
(amt). There are five constraints that those input events 
must meet before a transfer can take place: 

• c1 : src must be a valid account 

• c2 : tgt must be a valid account 

• c3 : amt must be greater than zero 

• c4 : amt must be less than or equal to the balance 
of src 

• c5 : the bank must allow transfers from src to tgt 

[0015] The service should check whether or not a 
constraint is violated as soon as possible; hence, it must 
be prepared to react when only a subset of the three 
events is present. The service must then prompt for the 
remaining events. 

[0016] Some basic principles are now considered 
related to the design and implementation of interactive 
services with multiple and varied user interfaces. These 
principles are especially important in the presence of 
multiple interfaces including those based on spoken 
natural language. 

• Lesser order = greater freedom 

[001 7] The source account, target account, and dol- 
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lar amount of the transfer capability should be inde- 
pendent events that can be collected in any order, 
where all three events are necessary to perform a trans- 
fer. By making explicit independent and dependent 
events, it in clear what events may be reordered without 5 
affecting the behavior of the service. This points the way 
to our first principle of the service specification: 

• Principle 1: Specify the service logic as a set of 
constraints on events and introduce a constraint w 
between two events only when absolutely neces- 
sary for the correct functioning of a service. That is, 

the service logic should be able to accept inde- 
pendent input events in different orders. 

15 

• Eliminate errors oniy 

[0018] It is often desirable for a service to respond 
as soon as possible with error conditions about user 
input. Since a constraint on the input events may refer to 20 
any arbitrary subset of those events, it is desirable that 
the service logic should be able to accept arbitrary sub- 
sets of events at any time. 

• Principle 2: The service logic should accept an 25 
incomplete input, i.e., subsets of the universe of 
possible events. 

• / take that back! 

30 

[0019] Unfortunately, humans often change their 
mind and/or make mistakes. Whenever possible, serv- 
ices must accommodate these shortcomings of our 
species, providing a capability to correct information or 
back out of a transaction. This leads to our third princi- 35 
pie: 

• Principle 3: The service logic should allow the user 
to back up to correct or update previously submitted 
information at any time. 40 

A closely related principle is: 

• Principle 4: The service logic should allow the user 

to back up to previous points in the service. 45 

Ready or not? 

[0020] Services that obey the above principles gen- 
eralize from linear user interactions to potentially allow 50 
multiple points of interaction to be enabled at a given 
instant. This information serves as an abstraction of the 
current control point of the service, and can be handled 
in a different manner by different user interfaces. For 
example, in automatic speech recognition interfaces, 55 
the information about currently enabled events is used 
by the user interface in two ways: 



• to appropriately prompt the user for information, 
thus compensating for the lack of visual cues; and 

• to effectively parse the information provided by the 
user. 

[0021] A user interface need not respond to all cur- 
rently enabled events of the service. Thus, different user 
interfaces can formulate different queries to the user 
even though the control point in the underlying service 
logic, as revealed by the current set of enabled events, 
is the same. The decoupling that we are seeking 
between the user interface and the service logic, there- 
fore points the way to our last principle of the service 
specification: 

• Principle 5: At any point in the service, the service 
logic must automatically report to the user inter- 
faces all currently enabled events, user prompts, 
help, and ways to revert back to previous points in 
the service. 

[0022] User interfaces have two main responsibili- 
ties with respect to the SISL architecture that reflect the 
two-way information flow between the user interface 
and service logic: 

• Based on the events received from the service 
logic, via the service monitor, prompt the user to 
provide the appropriate information and respond if 
the user requests help. 

• Collect the information from the user and transform 
the information into events to be sent to the service 
logic, via the service monitor. 

[0023] Indeed, any user interface (Ul) that performs 
these functions can be employed in conjunction with an 
SISL service logic. Additionally, SISL provides a con- 
venient framework for designing and implementing web- 
based, applet-based, automatic speech recognition- 
based and telephone voice-based interfaces. To imple- 
ment such user interfaces, the Ul designer need only 
specify two functions corresponding to the prompt and 
help mechanism. For automatic speech recognition 
interfaces, a set of speech grammars, i.e., the input to a 
speech recognition engine that permits it to efficiently 
and effectively recognize spoken input, together with a 
third function that specifies which grammars to enable is 
also required. 

[0024] The required functions are: 

• A prompt function that generates the string to be 
given as the prompt to the user. The SISL infra- 
structure automatically causes automatic speech 
recognition-based interfaces to speak the prompt 
string. Web -based interfaces automatically display 
the prompt string, as well as, radio buttons corre- 
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sponding to the possible transaction choices. For 
the other prompt events, text fields are automati- 
cally displayed, while submit buttons are automati- 
cally displayed for enabled events that allow the 
user to return to earlier points in the service. 5 

• A help function that generates the string to be given 
as the prompt to the user. 

• A grammar function that enables the correct set of w 
grammar rules; this function is only needed for 
automatic speech recognition-based interfaces. 

[0025] From prescribed functions and grammars, 
the SISL infrastructure automatically coordinates the 15 
collection and event transformation mechanisms, and 
integrates the user interface with the service logic and 
the service monitor. For automatic speech recognition- 
based interfaces, the SISL infrastructure automatically 
generates a desktop interface based on JAVA Speech 20 
API. To enable telephone-based voice access to the 
service, SISL automatically generates VoxML pages, 
which specify the voice dialog to be carried out on a 
telephony platform. For Web-based interfaces, the SISL 
infrastructure automatically generates HTML (Hypertext 25 
Markup Language) pages. It is noted that SISL provides 
a mechanism for the Ul designer to customize the look 
and feel of the interface. 

[0026] FIG. 2 shows, in simplified block form, 
details of the SISL (Several Interfaces, Single Logic) 30 
architecture employed in the embodiment of FIG. 1 . The 
SISL architecture together with the event communica- 
tion protocol provides modularity between the service 
logic 201 and user interfaces 204. In particular, two fea- 
tures of the event communication protocol allow service 35 
logic 201 to function completely without knowledge of 
the specifics of the individual user interfaces 204. These 
features are: (1) events are multicast from service logic 
201 via service monitor 202 to user interfaces 204 and, 
consequently, service logic 201 does not need to know 40 
the destinations of these events; and (2) the source of 
the events from the user interfaces 204 is not recorded 
and, consequently, the service logic 201 does not know 
which one or more of user interfaces 204 sent the 
events. Service monitor 202 is responsible for maintain- 45 
ing this communication protocol. This modularity allows 
service providers to provide interchangeable user inter- 
faces 204, or add new ones, to a single consistent 
source of service logic and data. 

[0027] Specifically, shown in FIG. 2 are service 50 
logic unit 201, service monitor unit 202 and user inter- 
faces 204-1 through 204-N. The key principle underly- 
ing SISL is that all user interfaces 204 to a service share 
a single service logic 201 . All communications between 
the service logic 201 and its multiple user interfaces 204 55 
are through events, via a service monitor 202. Events 
from the service logic 201 are broadcast to the user 
interfaces 204 via 203 to the service monitor 202 and, 



then, via 205 as a set of enabled events to the user 
interfaces 204. At the outset of the service, for example 
the Any-Time Teller banking service, each user inter- 
face 204 registers with the service monitor 202 to indi- 
cate which events it is interested in receiving. After 
collecting information from the user, the user interfaces 
204 send events to the service monitor 202 via bi-direc- 
tional links 205; the service monitor 202 does not record 
the source of these events. The service monitor 202 
passes the events, one at a time, via 203 to the service 
logic 201 . 

[0028] Event communication supports decoupling 
of the service logic 201 and the user interfaces 204, and 
allows service providers to provide interchangeable 
user interfaces 204, or add new ones, to a single con- 
sistent source of service logic 201 and data. 
[0029] In each round of interaction, the SISL infra- 
structure automatically sends out a set of events via 203 
from the service logic 201 to the service monitor 202, 
corresponding to the events that are currently enabled 
in the service logic 201 . There are three kinds of events: 
prompt events, up events, and notify events. Prompt 
events indicate to the user interface what information to 
communicate to the user and what information the serv- 
ice is ready to accept. There are three kinds of prompt 
events: 

• prompt_choice events are disjunctive choices cur- 
rently enabled in the service logic 201. For exam- 
ple, after the user has successfully logged into the 
Any-Time Teller banking service, a choice among 
the different transaction types is enabled. The serv- 
ice logic 201 sends a prompt_choice_deposit, 
prompt_choice_withdrawal event, and a 
prompt_choice_transfer event, and so forth, via 203 
to the service monitor 202. 

• prompt_req events are the events currently 
required by the service logic 201 . For example, sup- 
pose the user has chosen to perform a transfer 
transaction. The Any-Time Teller requires that the 
user input a source account, a transfer account, 
and amount, and hence sends prompt_req_src, 
prompt_req_tgt, and prompt_req_amt via 203 to 
the service monitor 202. 

• prompt_opt events are events enabled in the serv- 
ice logic 201 , for which the user may correct previ- 
ously given information. For example, suppose the 
user is performing a transfer and has already pro- 
vided his/her source and target accounts, but not 
the amount. The service logic 201 sends prompt 
_opt_src, prompt_opt_tgt, and prompt_req_amt 
events via 203 to the service monitor 202. This indi- 
cates that the user may override the previously 
given source and target accounts with new informa- 
tion. 
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[0030] Up events correspond to prior points in the 
service logic 201 to which the user may go back. For 
example, the service logic 201 sends an up_MainMenu 
event via 203 to the service monitor 202. This allows the 
user to abort any transaction and go back up to the main 
menu. 

[0031] Notify events are simply notifications that the 
user interface 204 should give the user; for example, 
that a transaction has completed successfully or that 
information provided by the user was incorrect or incon- 
sistent. 

[0032] Thus, the service monitor 202 is essentially 
a controllable interface that manages the communica- 
tion between the service logic 201 and the multiple user 
interfaces 204. As such, service monitor 202 provides a 
communication mechanism in the form of a so-called 
"look ahead" in which the user can input information 
beyond what is currently required by the interactive 
service. The so-called look ahead is realized because 
the user interfaces 204 can supply information to serv- 
ice monitor 202 that is beyond, i.e., in addition, to that 
which is currently being requested by service logic 201 . 
Thus, service monitor 202 may already have informa- 
tion from user interfaces 204 that may be later 
requested by, but is not currently being requested by, 
service logic 201. Note that service monitor 202 does 
not request any further information from user interfaces 
204 until it does not have any information, i.e., input 
event, that matches an enabled event from service logic 
201. 

[0033] In summary, service monitor 202 is essen- 
tially a controllable memory unit including a bidirectional 
communication link 203 to service logic 201 and bidirec- 
tional communication links 205 to user interfaces 204. 
Service monitor 202 at any current instant stores a cur- 
rent set of user events from user interfaces 204, old 
user events not yet sent to service logic 201 and a cur- 
rent set of enabled events from service logic 201 . 
[0034] FIG. 3 is a flowchart illustrating the steps in 
implementing the operation of service monitor 202 in 
accordance with the invention. Accordingly, the process 
is started in step 301. Then, step 302 causes monitor 
202 (FIG. 2) to wait for a set of enabled events from 
service logic 201. Step the causes a set of enabled 
events to be sent from service monitor 202 to user inter- 
faces 204. Service monitor 202 is caused by step 304 to 
wait for a set of events corresponding to user inputs 
from user interfaces 204. Then, step 305 tests to deter- 
mine whether any remaining user event matches some 
enabled event. If the test result in step 305 is NO, con- 
trol is returned to step 303 and steps 303 through 305 
are iterated until step 305 yields a YES result. Upon 
step 305 yielding a YES result, step 306 causes a 
matching user event to be selected in priority order, the 
removal of the selected matching user event from the 
set of user events, and sending the selected user event 
to service logic 201. Then, step 307 causes service 
monitor 202 to wait for a set of enabled events from 



service logic 201 . Thereafter, control is returned to step 
305 and, then, appropriate ones of steps 303 through 
307 are iterated. 

[0035] FIG. 4 is a flow diagram illustrating a reactive 
5 constraint graph for a portion of the Any-Time Teller 
banking service example. 

[0036] In SISL, the service logic 201 (FIG. 2) of an 
application is specified as a reactive constraint graph, 
which is a directed acyclic graph with an enriched struc- 

w ture on the nodes. The traversal of reactive constraint 
graphs is driven by the reception of events from the 
environment; these events have an associated label, 
i.e., the event name, and may carry associated data. In 
response, the graph traverses its nodes and executes 

15 actions; the reaction of the graph ends when it needs to 
wait for the next event to be sent by the environment. 
For example, FIG. 4 shows a SISL reactive constraint 
graph that implements part of the functionality of the 
Any-Time Teller. 

20 [0037] Reactive constraint graphs can have three 
kinds of nodes, namely, choice nodes, constraint nodes 
and action nodes. 

[0038] Choice nodes represent a disjunction of 
information to be received from the user. Every choice 

25 node has a specified set of events. There are two forms 
of choice nodes, namely, event-based and data-based. 
Every event-based choice node has a specified set of 
events. For every event in this set, the SISL infrastruc- 
ture automatically sends out a corresponding 

30 prompt_choice event from the service logic 201 to the 
user interface, via the service monitor 202. The choice 
node waits for the user interface to send, via the service 
monitor 202, any event in the specified set. When such 
an event arrives, the corresponding transition is taken, 

35 and control transfers to the descendent, i.e., child, node. 
[0039] For example, if a startTransfer event arrives 
when control is at the choice node, the corresponding 
transition is taken and control is transferred to the target 
constraint node. To ensure determinism, all outgoing 

40 transitions of a choice node must be labeled with dis- 
tinct event names. 

[0040] Every data-based choice node has a speci- 
fied set of preconditions on data. To ensure determin- 
ism, these preconditions must be specified so that 

45 exactly one of them is "true" in any state of the system. 
When control reaches a data-based choice node, the 
transition associated with the unique "true" precondition 
is taken, and control is transferred to the child node. 
[0041] Constraint nodes represent a conjunction of 

so information to be received from the user. Every con- 
straint node has an associated set of constraints on 
events. Constraints have the following components: 

• The signature is the set of events occurring in the 
55 constraint. 

• The evaluation function is a boolean function on the 
events in the signature. 

• The optional satisfaction tuple consists of an 
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optional action, not involving user interaction, and 
an optional notify function that may return a notify 
event with an associated message. If the constraint 
evaluates to true, the action is executed, the notify 
function is executed and the returned notify event is 5 
sent to the user interface via the service monitor 
202. 

• The optional violation tuple consists of an optional 
action, not involving user interaction, an optional 
notify function that may return a notify event with an 10 
associated message, an optional uplabel function 
that may return the uplabel of an ancestor node and 

an optional child node. If the constraint evaluates to 
"false", the action is executed, the notify function is 
executed and the returned notify event is sent to the 15 
user interfaces 204via the service monitor 202. The 
uplabel function is also executed; if it returns an 
ancestor's uplabel, it is generated, and hence con- 
trol reverts back to the ancestor node. If no ances- 
tor node's uplabel is returned and a child node is 20 
specified, control is transferred to the specified 
child node. 

[0042] For example, amt<= Balance (src) of 407 in 
FIG. 4 is equivalent to ?amt<=Balance(?src), and is the 25 
evaluation of a prescribed constraint. The signature of 
this constraint is the set {amt, src], and the satisfaction 
notify function and violation notify function, respectively, 
report to the user whether or not the source account has 
enough funds to cover the requested amount. The noti- 30 
fication ?EventName refers to the data on the event with 
name EventName. 

[0043] Every constraint node has the following com- 
ponents: 

35 

• An associated set of constraints. In the current 
semantics and implementation, this set is totally 
ordered, specifying the priority order in which the 
constraints are evaluated. 

40 

• An optional entry action, not involving user interac- 
tion. 

• An optional finished tuple, consisting of an optional 
exit action, not involving user action, an optional 45 
notify function, an optional uplabel function and an 
optional child node. 

[0044] The SISL infrastructure automatically sends 
out a prompt_req event, from the service logic 201 (FIG. 50 
2) to the user interfaces 204via the service monitor 202, 
for every event that is still needed in order to evaluate 
some constraint. Additionally, the constraint node sends 
out a prompt_opt event for all other events mentioned in 
the constraints. These correspond to the information 55 
that can be corrected by the user. 
[0045] In every round of interaction, the constraint 
node waits for the user interface to send, via the service 



monitor 202, any event that is mentioned in the signa- 
ture of its associated constraints. Each constraint asso- 
ciated with a constraint node is evaluated as soon as all 
of its events have arrived. If an event is resent by the 
user interfaces, i.e., information is corrected, all con- 
straints with that event in their signature are re-evalu- 
ated. For every evaluated constraint, its optional 
satisfied/violated action is automatically executed, and 
a notify event is automatically sent to the user inter- 
faces, with the specified message. 
[0046] Specifically, the constraints are evaluated in 
the specified priority order, currently the total ordered 
set. If any constraint is violated, the last received event 
is automatically erased from all constraints, since it 
caused an inconsistency. Furthermore, the violation 
action is executed, the notify function is executed and 
the returned notify event is automatically sent to the 
user interfaces 204via the service monitor 202. The 
uplabel function is also executed; if it returns an ances- 
tor's uplabel, it is generated and, hence control reverts 
back to that ancestor. For example, in FIG. 4, the con- 
straint node 403 for the transfer capability checks 
whether the source account is an active account of the 
given customer, i.e., user, via a prescribed constraint. If 
not, it generates the uplabel "LoginMenu ,l , and control is 
transferred back to the Login node 401. Then, the user 
must re-enter his/her login. If no ancestor node's upla- 
bel is returned and a child node is specified, control is 
transferred to that child node. For example, in FIG. 4, 
the Login node 401 checks whether the login is of a cus- 
tomer in good standing, via a constraint 
good_customer, which evaluates goodCustomer(/og/n). 
If not, control is transferred to the Quit node and the 
service is terminated. If no ancestor node's uplabel is 
returned or child node specified for the violated con- 
straint, the node reverts to waiting for events to arrive. 
[0047] If no constraints have been violated, the 
action of every satisfied constraint is executed, the 
associated notify functions are executed and the 
returned notify events are sent to the user interfaces 
204via the service monitor 202. 
[0048] When all the constraints have been evalu- 
ated and are satisfied, the exit action and notify function 
associated with the constraint node are executed and 
the returned notify event is sent to the user interfaces 
204via the service monitor 202. The uplabel function is 
also executed; if it returns an ancestor node's uplabel, it 
is generated, and hence control is returned back to the 
ancestor node. If no ancestor node's uplabel is returned 
and a child node it specified, control is transferred to 
that child node. 

[0049] Action nodes represent some action, not 
involving user interaction, to be taken. After the action is 
executed, control transfers to the child node. 
[0050] Nodes can have an optional "uplabel", which 
is used to transfer control from some child node back up 
to the node, allowing the user to revert back to previous 
points in the service. In each round of interaction, the 
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SISL infrastructure automatically sends out an up 
event, from the service logic 201 to the user interfaces 
204via the service monitor 202, corresponding to the 
uplabel of every ancestor of the current node. 
[0051] Nodes can also have a self-looping arc, with 5 
a boolean precondition on data. This indicates that the 
subgraph from the node will be repeatedly executed 
until the precondition becomes false. 
[0052] An example execution of the Any-Time Teller 
Banking Service is shown in FIG. 4. The service initially 10 
causes the statement "Welcome to Any-Time Teller" to 
be spoken. The control point is at the root node, which 
is a constraint node. For the constraint of the root node 
to be satisfied, the login and pin values must be identi- 
fied, i.e., iogin==pin, as shown in step 401 of FIG. 4. 15 
The SISL infrastructure automatically sends out a 
prompt_req_login and a prompt req_pin from the serv- 
ice logic 201 to the user interfaces, via the service mon- 
itor 202. The user interfaces, via the prompt function, 
respond by saying "Please specify your login and per- 20 
sonal identification number". For the Web-based user 
interface, text fields for the login and pin are automati- 
cally generated, in known fashion; forthe speech recog- 
nition-based user interface, the grammars specified in 
the grammar function are automatically enabled. 25 
[0053] In this example, suppose that the user states 
"My login is Mary Smith and my pin is Mary Smith", and 
hence a login event with the value "Mary Smith" and a 
pin event with the value "Mary Smith" are sent to the 
service logic 201 . Since the login and pin are identical, 30 
the constraint is satisfied. The SISL infrastructure auto- 
matically sends out a notify event with the message 
"Hello Mary Smith". Welcome to the SISL Any-Time 
Teller". The user interface makes this statement to the 
user. 35 

• Login successful 

[0054] Control now proceeds to step 402 and to the 
choice node. The SISL infrastructure automatically 40 
sends out: 

• prompt_choice_startDeposit', 

• prompt_choice startWithdrawal; 

• prompt_choice_startTransfer; and 45 

• prompt_choice_userQuit; 

events from the service logic 201 (FIG. 2) to the user 
interfaces 204, via the service monitor 202, correspond- 
ing to the enabled choices. The user interfaces 204ask 50 
the user "What Transaction would you like to do?" For 
an automatic speech recognition-based user interface, 
if the user states "I need help", the user interface states, 
via a help function, "You can make a withdrawal, deposit 
transfer or you can quit the service". Consider that the 55 
user now chooses to perform a transfer, the 
startTransfer event is sent to the service logic 201 . 



• Transfer 

[0055] Control now proceeds to constraint node 
406. The SISL infrastructure automatically sends out: 

• prompt_req_src; 

• prompt_req_tgt\ and 

• prompt _req_amt 

events from the service logic 201 to the user interfaces 
204, via the service monitor 202, together with a 
up_MainMenu event, since it is the uplabel of an ances- 
tor node. Assume that the user respond with "I would 
like to transfer One Thousand Dollars ($1 ,000.00)from 
my checking account", orequivalently "From checking, I 
would like to transfer One Thousand Dollars 
($1,000.00)". Either order of the transfer request infor- 
mation is allowed; furthermore, this information in par- 
tial, since the target account is not specified. The user 
interface 204 sends a src event and an amt event, with 
the corresponding data, to the service monitor 202, 
which sends them one at a time to the service logic 201 . 
Assume that the src event is sent first, followed by the 
amt event. The constraints amt>o, isValid(src) and 
amt<= Balance (src) are automatically evaluated. 
[0056] Assume that the checking account does not 
have a balance of at least $1,000.00; hence, there is a 
constraint violation and the supplied information is 
erased, since it was sent last. Note that constraints are 
evaluated as soon as possible; for example, the user is 
not required to specify a target account in order for the 
balance on the source account to be checked. The SISL 
infrastructure then automatically sends out a 
prompt_opt_src, prompt_req_tgt, prompt_req_amt and 
upMainmemu events from the service logic 201 to the 
user interfaces, via the service monitor 202, as well as, 
a notify event with the message "Your checking account 
does not have sufficient funds to cover the amount of 
$1,000.00. Please specify an amount and a target 
account." The user interface 204 then notifies the user 
with this message and prompts the user for the informa- 
tion. 

[0057] Assume now that the user states "Transfer 
Five hundred Dollars ($500.00) to savings". The amt 
and tgt events are sent to the service monitor 202, and 
passed to the service logic 201. The constraints are 
now all evaluated and satisfied, the service logic 201 
automatically sends a notify event to the user interfaces 
204 with the message. "Your transfer of $500.00 from 
checking to savings was successful". 
[0058] Control then is returned back up to the 
choice node 402; the loop on the incoming arc to the 
choice node indicates that the corresponding subgraph 
is repeatedly executed until the condition on the arc 
becomes false. If the user wants to quit the service, the 
userQuit event is sent to the service logic 201, the 
hasQuit variable is set to true, and the loop is termi- 
nated. While in step 402, a loop step 404 is performed 
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to test if the user has terminated, i.e., quit, the session. 

• Back up to the Main Menu 

[0059] If the user would like to abort at any time dur- 
ing a withdrawal, deposit or transfer transaction, he/she 
can state "I would like to go back up to the Main Menu", 
which results in an up_MainMenu event to be sent to 
the service logic 201 . This causes control to be returned 
to the choice node 402, which has an associated 
upMainMenu label. 

• Make a Deposit 

[0060] If a user wishes to make a deposit control 
proceeds to the deposit constraint node 406. The SISL 
infrastructure automatically sends out 

• prompt_req_tgt, 

• prompt _req_amt 

events from the service logic 201 to the user interfaces 
204 via service monitor 202. If the target account is valid 
and the amount is greater than zero (0) the deposit is 
made and the associated notification is executed. 

• Make a Withdrawal 

[0061 ] If a user wishes to make a withdrawal control 
proceeds to the deposit constraint node 407. The SISL 
infrastructure automatically sends out: 

• prompt_ req_src , 

• prompt_req_amt 

events from the service logic 201 to the user interfaces 
204 via service monitor 202. If the source account is 
valid and the amount is greater than zero (0), it is deter- 
mined if amt<=Balance(src) and, if so, the withdrawal is 
made and the associated notification is executed. 
[0062] FIG. 5 illustrates the steps performed in an 
example execution of the Any-Time Teller including look 
ahead in accordance with the invention. 
The service is started in an INITIAL STATE OF 
SERVICE. Then, in this example, the following steps 
are effected: 

• SERVICE LOGIC 201: sends to Service Monitor 
202 a set of enabled events. prompt_req events: 
{login, pin}. 

• SERVICE MONITOR 202: sends set of enabled 
events to User Interfaces 204. 

• USER INTERFACES 204: prompt the user with 
"Welcome to the Any-Time Teller banking service. 
Please specify your login and personal identifica- 
tion number". 



• USER: states "I want to transfer money from my 
checking account. My login is JohnDoe and my pin 
is 4341". 

5 ♦ USER INTERFACES 204: send to Service Monitor 
202 the set of user events: {startTransfer, 
src(checking), login(JohnDoe), pin(4341). 

• MATCHING EVENTS: {login, pin}. 

10 

• SERVICE MONITOR: sends login(JohnDoe) event 
to Service Logic. 

• SERVICE LOGIC 201: sends to Service Monitor 
15 202 set of enabled events. prompt_req events: 

{pin}, prompt_opt events: {login}. 

• REMAINING USER EVENTS: {startTransfer, 
src(checking), pin(4341)}. 

20 

• MATCHING EVENTS: {pin}. 

• SERVICE MONITOR 202: send pin(4341) event to 
Service Logic 201 . 

25 

• SERVICE LOGIC 201: sends to Service Monitor 
202 set of enabled events. prompt_choice events: 
{start Deposit, startWithdrawal, startTransfer, user- 
Quit}. 

30 

• REMAINING USER EVENTS: {startTransfer, 
src(checking)}. 

• MATCHING EVENTS: {startTransfer}. 

35 

• SERVICE MONITOR 202: sends startTransfer 
event to Service Logic 201 . 

• SERVICE LOGIC 201: sends to Service monitor 
40 202 set of enabled events. prompt_req events: {src, 

tgt, amt}. 

• REMAINING USER EVENTS: {src(checking)}. 
45 • MATCHING EVENTS: {src}. 

• SERVICE MONITOR 202: sends src(checking) 
event to Service Logic 201 . 

50 • SERVICE LOGIC 201: sends to Service Monitor 
202 a set of enabled events. prompt_req events: 
{tgt, amt}. prompt_opt events: {src} 

• SERVICE MONITOR 202: sends set of enabled 
55 events to User Interfaces 204. 

• USER INTERFACES 204: prompt user with "Hello 
JohnDoe. Please specify the target account and the 
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amount you would like to transfer". 

[0063] The above-described embodiment is, of 
course, merely illustrative of the principles of the inven- 
tion. Indeed, numerous other methods or apparatus 5 
may be devised by those skilled in the art without 
departing from the spirit and scope of the invention. 

Claims 

10 

1. Apparatus for use in providing an interactive user 
service including, 

a service logic unit, and 

a plurality of user interfaces, 15 
the apparatus being CHARACTERIZED BY 
a service monitor unit for controlling communi- 
cation of information between said service 
logic unit and a plurality of user interfaces, at 
least one of said user interfaces having natural 20 
language understanding, and said service 
monitor unit including 

receiver means adapted to receive information 
from any of said plurality of user interfaces and 
adapted to receive requests for information 25 
from said service logic unit, and 
correlator means adapted to match received 
information from said plurality of user interfaces 
to requested information from said service 
logic unit and to supply the matched received 30 
information to said service logic unit in a pre- 
scribed manner. 

2. The apparatus as defined in claim 1 CHARACTER- 
IZED IN THAT said service logic unit provides data 35 
and logic to said plurality of user interfaces and 
includes service logic means adapted to facilitate at 
least different ordering of inputs from any of said 
user interfaces, and/or incomplete information from 
any of said user interfaces, and/or correction of 40 
information from any of said user interfaces, and/or 
returning of control to prior points in said interactive 
service in response to a request from at least one of 
said plurality of user interfaces. 

45 

3. The apparatus as defined in claim 1 and 2 CHAR- 
ACTERIZED IN THAT each of said user interfaces 
includes means adapted to allow users to supply 
information to said service monitor unit in addition 

to information currently requested by said service so 
logic unit. 

4. The apparatus as defined in claims 1 and 2 CHAR- 
ACTERIZED IN THAT said communication of infor- 
mation is of events. 55 

5. The apparatus as defined in claim 4 CHARACTER- 
IZED IN THAT said requested information is a set of 



enabled events. 

6. The invention as defined in claim 5 CHARACTER- 
IZED IN THAT said service monitor unit includes 
means adapted to receive a set of input events from 
at least one of said user interfaces and means 
adapted to controllably supply said received input 
events in a prescribed order to said service logic 
unit. 

7. The invention as defined in claim 6 CHARACTER- 
IZED IN THAT said service logic includes means 
adapted to controllably generates a set of enabled 
events in response to said supplied events and 
means adapted to supply said set of enabled 
events to said service monitor unit, and said service 
monitor unit includes means adapted to supply said 
set of enabled events in a prescribed fashion to all 
appropriate ones of said user interfaces. 

8. The apparatus as defined in claim 5 CHARACTER- 
IZED IN THAT said information, received by said 
receiver means is a set of input events, and said 
correlator means includes means adapted to match 
said received input events to said enabled events 
and means is adapted to supply said matched input 
events one at a time to said service logic unit in a 
prescribed order. 

9. The apparatus as defined in claim 8 CHARACTER- 
IZED IN THAT said prescribed order is a prescribed 
priority order. 

10. The apparatus as defined in claim 9 CHARACTER- 
IZED IN THAT said priority order is prescribed by 
the service provider. 

11. The apparatus as defined in claim 10 CHARAC- 
TERIZED IN THAT said service logic unit includes 
means adapted to supply a new set of enabled 
events to said service monitor unit in response to 
an input event from said service monitor unit, and 
said service monitor unit includes means adapted 
to notify said user interfaces to send a new set of 
events. 

12. The invention as defined in claim 1 1 CHARACTER- 
IZED IN THAT said service monitor unit includes 
transmitter means adapted to transmit said notifica- 
tion to said user interfaces instruction said user 
interfaces to send a new set of input events to said 
service monitor unit only when said correlator 
means indicates that no current input event 
matches an enabled event. 

13. The invention as defined in claim 12 CHARACTER- 
IZED IN THAT said transmitter means is adapted to 
transmit said set of enabled events to said user 
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interfaces from said service monitor unit only when 
said correlator means indicates that no current 
input event matches an enabled event. 

14. A method for providing an interactive user service 
comprising a service logic unit, a service monitor 
unit and a plurality of user interfaces including the 
step of, 

controlling communication of information 
through said service monitor unit between said 
service logic unit and said plurality of user 
interfaces, 

the method being CHARACTERIZED IN THAT 
at least one of said user interfaces has natural 
language understanding, and said step of con- 
trolling communication includes the steps of 
receiving information from any of said plurality 
of user interfaces, 

receiving requests for information from said 
service logic unit, 

matching received information to requested 
information, and 

supplying the matched received information to 
said service logic unit in a prescribed manner. 

15. The method as defined in claim 14 CHARACTER- 
IZED BY providing data and logic from said service 
logic unit to said plurality of user interfaces and 
facilitating at least different ordering of inputs from 
any of said user interfaces, and/or incomplete infor- 
mation from any of said user interfaces, and/or cor- 
rection of information from any of said user 
interfaces, and/or returning of control to prior points 
in said interactive service in response to a request 
from at least one of said plurality of user interfaces. 

16. The method as defined in claims 14 and 15 CHAR- 
ACTERIZED BY a step of supplying information 
from users of said user interfaces to said service 
monitor unit in addition to information currently 
requested by said service logic unit. 

17. The method as defined in claims 14 and 15 CHAR- 
ACTERIZED IN THAT said communication of infor- 
mation is of events. 

18. The method as defined in claim 17 CHARACTER- 
IZED IN THAT said requested information is a set of 
enabled events. 

19. The method as defined in claim 18 CHARACTER- 
IZED BY the steps of receiving a set of input events 
from at least one of said user interfaces and con- 
trollably supplying said received input events in a 
prescribed order to said service logic unit. 

20. The method as defined in claim 19 CHARACTER- 



IZED BY the steps of controllably generating a set 
of enabled events in response to said supplied input 
events, supplying said set of enabled events to said 
service monitor unit, and supplying said set of ena- 
5 bled events from said service monitor unit in a pre- 

scribed fashion to all appropriate ones of said user 
interfaces. 

21. The method as defined in claim 18 CHARACTER- 
10 IZED IN THAT said received information is a set of 

input events, and said step of matching includes 
matching said received input events to said enabled 
events and said step of supplying supplies said 
matched input events one at a time to said service 
15 logic unit in a prescribed order. 

22. The method as defined in claim 21 CHARACTER- 
IZED IN THAT said prescribed order is a prescribed 
priority order. 

20 

23. The method as defined in claim 22 CHARACTER- 
IZED IN THAT said priority order is prescribed by 
the service provider. 

25 24. The method as defined in claim 23 CHARACTER- 
IZED BY a step of notifying said user interfaces to 
send a new set of events. 

25. The method as defined in claim 24 CHARACTER- 
30 IZED IN THAT said step of notifying transmits a 

notification to said user interfaces to transmit a new 
set of input events to said service monitor unit only 
when said step of matching indicates that no cur- 
rent input event matches an enabled event. 

35 

26. The method as defined in claim 25 CHARACTER- 
IZED BY a step of transmitting said set of enabled 
events to said user interfaces from said service 
monitor unit only when said step of matching indi- 

40 cates that no current input event matches an ena- 
bled event. 
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S FIG. 5 

INITIAL STATE OF SERVICE 

SERVICE LOGIC: SENDS TO SERVICE MONITOR SET OF ENABLED EVENTS. 
prompLreq events: {login, pin| 

SERVICE MONITOR: SENDS SET OF ENABLED EVENTS TO USER INTERFACES 

USER INTERFACES: PROMPT THE USER WITH "WELCOME TO THE ANY-TIME TELLER 
BANKING SERVICE. PLEASE SPECIFY YOUR LOGIN AND PERSONAL IDENTIFICATION NUMBER". 

USER: STATES "I WANT TO TRANSFER MONEY FROM MY CHECKING ACCOUNT. 
MY LOGIN IS JohnDoe AND MY PIN IS 4341" 

USER INTERFACES: SEND TO SERVICE MONITOR THE SET OF USER EVENTS: 
{startTransfer, src(checking), login(JohnDoe), pin(4341) 

MATCHING EVENTS: {login, pin} 

SERVICE MONITOR: SENDS login(JohnDoe) EVENT TO SERVICE LOGIC 

SERVICE LOGIC: SENDS TO SERVICE MONITOR SET OF ENABLED EVENTS. 
prompLreq events: {pin}, prompLopt events: {login} 

REMAINING USER EVENTS: {startTransfer, src(checking), pin(434l)| 
MATCHING EVENTS: {pin} 

SERVICE MONITOR: SEND pin(4341) EVENT TO SERVICE LOGIC 

SERVICE LOGIC: SENDS TO SERVICE MONITOR SET OF ENABLED EVENTS. 
prompLchoice events: {startDeposit, startWithdrawal, startTransfer, userQuitj 

REMAINING USER EVENTS: {startTransfer, src(checking)} 

MATCHING EVENTS: {startTransfer} 

SERVICE MONITOR: SENDS START TRANSFER EYENT TO SERVICE LOGIC 

SERVICE LOGIC: SENDS TO SERVICE MONITOR SET OF ENABLED EVENTS. 
prompLreq events: {src, tgt, amt} 

REMAINING USER EVENTS: {src(checking)| 

MATCHING EVENTS: {src} 

SERVICE MONITOR: SENDS src(checking) EVENT TO SERVICE LOGIC 

SERVICE LOGIC: SENDS TO SERVICE MONITOR SET OF ENABLED EVENTS. 
prompLreq events: {tgt, amt}, prompLopt events: {src} 

SERVICE MONITOR: SENDS SET OF ENABLED EVENTS TO USER INTERFACES 

USER INTERFACES: PROMPT THE USER WITH "HELLO JohnDoe. PLEASE SPECIFY THE 
^ TARGET ACCOUNT AND THE AMOUNT YOU WOULD LIKE TO TRANSFER". 
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